Video and audio delivery presents significant network transmission challenges with display failure possible at multiple points along a transmission pathway. For example, “on demand” streaming content is encoded for storage, transmitted electronically via the Internet, decoded on a device on a network, transmitted wirelessly to a display and then decoded by that display for playback. Errors can occur at one or more points within this transmission, making the exact source of the errors difficult to isolate. Most video display analysis systems can only be used with specific input sources, requiring different data collection systems for each type of input. For example, current testing methods typically require a hard medium such as a video card from which to collect data and objectively analyze image playback quality. However, client devices such as wireless displays do not have a hard medium that can be used to download playback information, requiring the use of subjective “golden-eye” (real-time human-based) testing in order to analyze image playback quality.
Generally, video quality is analyzed using (1) subjective determinations where human subjects score the quality of video playback, aka “golden-eye” testing, and (2) objective metrics such as Peak Signal to Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM), multi-scale structural similarity (MS-SSIM) and the Sarnoff Picture Quality Rating (PQR). While human perception is the desired standard, subjective testing requires more time, resources, and preparation than objective measures, making it difficult to use during the development phase of a video playback display or system. Additionally, subjective determinations rely on the statistical averaging of experiences from several individuals, requiring uniform random sampling across people from different backgrounds, cultures, attitudes, expectations, genders, ages, physical abilities, interest levels and the like, and is frequently difficult to reproduce. Subjective analysis is also unable to detect the specific time point of an error or the cause of the error, merely rating the overall playback on a scale of 1 (bad) to 5 (excellent) which makes it difficult to address any issues seen. Objective measures, while less expensive and easily replicable, frequently do not have stable correlations with subjective human perception.
Analyzing video quality is one of the biggest challenges when evaluating codecs, transmission systems, and client devices. Currently, companies use a combination of objective and subjective tests to validate playback displays; however, both of these methods have drawbacks. Additionally, as technology advances and the medium or system for content delivery changes, a gap is created in video analysis tools requiring the return to purely subjective analysis.